The present invention relates to an image recording apparatus and method for describing a virtual space on the basis of sensed images and, more particularly, to a method of efficiently adding location information associated with image sensing location to a sensed image sequence obtained by sensing an object using cameras. The present invention also relates to an image database system built to describe a virtual space on the basis of sensed images. Furthermore, the present invention relates to a recording medium that stores a computer program for implementing the above-mentioned recording control.
In recent years, attempts have been made to build civic environments where many people socially live in cyber spaces formed by computer networks. Normally such virtual spaces are described and displayed using conventional CG techniques. However, since CG expressions based on geographic models have limitations, Iamge-Based Rendering (IBR) based on sensed images has come to the forefront of the technology.
Reproducing sensed image data as they are amounts to merely experiencing what a photographer has gone through. For this reason, a technique for generating and presenting an arbitrary scene in real time using the IBR technique has been proposed. More specifically, when sensed images are processed as independent ones, and are re-arranged in accordance with a viewer""s request, the viewer can walk through his or her desired moving route at a remote place, and can feel a three-dimensional virtual space there.
It is effective for searching for and re-constructing a desired image in accordance with the viewer""s request to use the location information of points where the individual images were taken. That is, an image closest to the viewers request is selected from a database, and undergoes proper image interpolation so as to generate and display an optimal image.
FIG. 1 shows the principle of wide-area walkthrough using sensed images.
More specifically, sensed images are prepared for narrow areas 1, 2, 3, and 4. In order to implement wide-area walkthrough (e.g., along a route 10 or 11) that allows the viewer to walk across these narrow areas, an image in a space between adjacent narrow areas must be obtained by interpolation. When the viewer is currently located at a position between the narrow areas 2 and 3, and the space between these areas is obtained by interpolation, specific sensed images for the areas 2 and 3 must be obtained by a search on the basis of information associated with the current location of the viewer between the areas 2 and 3. In other words, in order to obtain required images by a search on the basis of the location information of the user, a database of sensed images must be prepared in advance in accordance with location data upon image sensing.
In order to attain precise interpolation and to smoothly connect the interpolated images and sensed images, as the viewer may walk through in the 360xc2x0 range around him or her, sensed images of the environment must be taken by a large number of cameras disposed to point in various directions, and an image database must be built using these sensed images.
In order to attain precise interpolation on the basis of images obtained using a plurality of cameras, the image sensing centers of many cameras must agree with each other. However, it is not easy to arrange many cameras in such way.
To solve this problem, conventionally, a plurality of mirrors are set symmetrically about a given point, and the mirror surfaces of the individual mirrors are set so that light beams coming from the surrounding portions are reflected upward, thereby setting the image sensing centers of the cameras at one point, as shown in FIG. 2.
However, in the prior art, in order to build a database of sensed images, location information is added in units of image frames of the individual sensed images.
The location information is added to each image in such a manner that image sensing is done while moving a video camera at a constant speed so as to successively record images on a video tape, and the position of the portion of interest is calculated on the basis of the distance from the beginning of that video tape to the position of that tape portion which stores the image to which the location information is to be added. Then, a sensed image database is formed by combining the images on the tape and the image sensing location information of these images.
Hence, suchworks are nearly manually done, and requires much time as the number of sensed images becomes larger.
In order to implement wide-area walkthrough, each sensed image must have a broad range (wide angle). When a wide-angle image is obtained by a single camera, if image interpolation is done using neighboring pixels of the wide-angle image, many errors are produced in an image obtained by interpolation, and continuity is substantially lost between the interpolated image and sensed image. When a plurality of cameras (n cameras) are set at wide-angle positions to take images, the addition work of the location information increases n times.
The present invention has been made in consideration of the above situation and has as its object to provide an image recording method and apparatus, which can efficiently add location information associated with each image sensing location to a sensed image sequence obtained by a camera.
It is another object of the present invention to provide a database apparatus which builds an image database to which time information is added so that location information associated with each image sensing location can be efficiently added to a sensed image sequence obtained by a camera.
In order to achieve the above objects, according to the present invention, a method of recording images obtained by sensing an object using a plurality of cameras which point in a plurality of discrete azimuth directions so as to obtain a basis for forming a sequence of three dimensional image spaces, is characterized by comprising the steps of:
storing, in a first memory, data frames of images generated by sequentially sensing an object using a camera of the plurality of cameras together with specifying information that specifies each data frame;
acquiring location information indicating an image sensing location of each data frame, and storing, in a second memory, the location information together with acquisition time information indicating an acquisition time when the location information is acquired; and
storing, in a third memory, a pair of specifying information that specifies each data frame, and acquisition time information corresponding to the location information acquired at the time when the data frame is generated.
Image data require a large memory capacity. When no high-speed processing is required, a sequential memory has a large capacity. Hence, according to a preferred aspect of the present invention, the first memory comprises a sequential memory, and the specifying information is a frame number indicating a recording order of the data frames in the first memory.
According to a preferred aspect of the present invention, the specifying information is information indicating a relative time upon recording the data frame in the first memory.
According to a preferred aspect of the present invention, the first memory comprises video tape memories arranged in units of cameras, and the second memory comprises a magnetic disk.
According to a preferred aspect of the present invention, the specifying information includes time codes generated by each camera. The time codes have different default values among the cameras. Hence, in this aspect, calibration is attained by calculating the difference between the time code, as a reference, of a specific one of the plurality of cameras, and that of another camera.
According to a preferred aspect of the present invention, the plurality of cameras are mounted on a vehicle.
According to a preferred aspect of the present invention, posture information indicating a posture of the camera is recorded together in the second memory.
A GPS sensor can provide location information with high precision. Hence, according to a preferred aspect of the present invention, the location information is obtained from a GPS sensor.
According to a preferred aspect of the present invention, the method further comprises the steps of:
reading out, using a value of the acquisition time information as a key, the specifying information corresponding to the acquisition time information from the third memory, and the location information corresponding to the acquisition time information from the second memory; and
recording the image data frame in the first memory specified by the specifying information read out from the third memory, and the location information read out from the second memory, in a fourth memory in association with each other.
The present invention is also directed to an image database formed by the above-mentioned recording method.
According to the present invention, the above objects are also achieved by a method of recording images obtained by sensing an object using a plurality of cameras which point in a plurality of discrete azimuth directions so as to obtain a basis for forming a sequence of three dimensional image spaces, comprising the steps of:
storing, in a first memory, data frames of images gene rated by sequentially sensing an object using a camera of the plurality of cameras together with generation time information that indicates a generation time of-each data frame; and
acquiring location information associated with an image sensing location of each data frame, and storing, in a second memory, the location information together with acquisition time information indicating an acquisition time of the location information.
According to a preferred aspect of the present invention, the image data frame in the first memory and the location information in the second memory, the generation time information and acquisition time information of which have equal values are recorded on a fourth memory in association with each other.
According to the present invention, the above objects are also achieved by an image database system obtained by sensing an object using a plurality of cameras which point in a plurality of discrete azimuth directions, comprising:
a location-time database including location information associated with an image sensing location together with first time information representing an acquisition time of the location information; and
an image-time database including image data together with second time information indicating a generation time of the image data.
In order to achieve the above objects, according to the present invention, a recording apparatus for recording images obtained by sensing an object using a plurality of cameras which point in a plurality of discrete azimuth directions so as to obtain a basis for forming a sequence of three dimensional image spaces, is characterized by comprising:
a first memory storing data frames of images generated by sequentially sensing an object using a camera of the plurality of cameras together with specifying information that specifies each data frame;
a second memory storing location information indicating an image sensing location of the camera together with acquisition time information indicating an acquisition time of the location information; and
a third memory storing a pair of specifying information that specifies each data frame, and acquisition time information corresponding to the location information acquired at the time of generation of the data frame.
According to a preferred aspect of the present invention, the first memory comprises a sequential memory, and the specifying information is a frame number indicating a recording order of the data frames in the first memory.
According to a preferred aspect of the present invention, the specifying information is information indicating a relative time upon recording the data frame in the first memory.
According to a preferred aspect of the present invention, the first memory comprises video tape memories arranged in units of cameras, and the second memory comprises a magnetic disk.
According to a preferred aspect of the present invention, the specifying information includes time codes generated by the cameras, and the apparatus further comprises means for using the time code of a specific one of the plurality of cameras as a reference time code, and calculating differences between the time code of the specific one camera, and time codes of other cameras.
According to a preferred aspect of the present invention, the plurality of cameras are mounted on a vehicle.
According to a preferred aspect of the present invention, posture information indicating a posture of the camera is recorded together in the second memory.
According to a preferred aspect of the present invention, the location information is obtained from a GPS sensor.
According to a preferred aspect of the present invention, the apparatus further comprises:
a reader unit reading out, using a value of the acquisition time information as a key, the specifying information corresponding to the acquisition time information from the third memory, and the location information corresponding to the acquisition time information from the second memory; and
a recorder unit recording the image data frame in the first memory specified by the specifying information read out from the third memory, and the location information read out from the second memory, in a fourth memory in association with each other.
Also, still another object of the present invention is achieved by a recording apparatus for recording images obtained by sensing an object using a plurality of cameras which point in a plurality of discrete azimuth directions so as to obtain a basis for forming a sequence of three dimensional image spaces, comprising:
a first memory storing data frames of images generated by sequentially sensing an object using a camera of the plurality of cameras together with generation time information that indicates a generation time of each data frame; and
a second memory storing location information associated with an image sensing location of each of the data frame together with acquisition time information indicating an acquisition time of the location information.
According to a preferred aspect of the present invention, the apparatus comprises a fourth memory which stores the image data frame in the first memory and the location information in the second memory, the generation time information and acquisition time information of which have equal values, in association with each other.
Other features and advantages of the present invention will be apparent from the following description taken in conjunction with the accompanying drawings, in which like reference characters designate the same or similar parts throughout the figures thereof.